The Jetstream Service API

Table of Contents

Service Description

Jetstream is a Jibo Platform Service that mediates a skill's interaction with the Hub (it replaces the ASRservice in Jibo V1). In this role it must handle requests from skills to recognize speech, it must engage onboard recognizers for HJ spotting, name learning and speaker ID, and it must pass along proactive trigger information from the skills to the Hub and back. The genesis of Jetstream's basic structure is visible in the Jetstream Flow Diagram at the end of this document.

Jetstream presents a URL-based API to skills via HTTP requests and Websocket messages. For the HTTP requests, the parameters are supplied as a JSON string in the body of a POST request to the given URL. Likewise the HTTP responses and Websocket messages supply their information to the skill as a JSON payload in the body of the response or message.

In this document, red colored text with a ⚠ symbol indicates a feature that should not be counted on due to being either not-yet-implemented or likely to change soon.

The Jetstream HTTP API

Herewith is a thematically arranged list of all the HTTP-based commands Jetstream offers.

Local Turns: Skill-driven ASR

A local turn is the means by which a skill requests speech recognition services. The word "local" here refers to a turn started by the skill, in contrast to turns started when a user says "HJ + phrase" that have a (global) reach beyond the skill. Since turns are long operations that involve waiting for user speech, they run asynchronously after they're requested. The skill will post a turn request to Jetstream, which is acknowledged without delay and which causes Jetstream to begin the process of connecting and sending audio to the Hub to perform speech recognition. As the turn progresses, Jetstream posts messages about its status or conclusion to /events.

Start Local Turn

Start a local turn, it remains active until it reports a status of SUCCEEDED, FAILED, INTERRUPTED, REJECTED, CANCELED, or TIMEDOUT via a TURN_RESULT message on the /events Websocket.

URL	/listen/start_local_turn
Parameters
name	type	Description
nluRules	string[]	Names of rules that should be activated for this turn. If this list is empty, the text output of the ASR step is not parsed for meaning and is simply returned to the skill.
ignoreHJ	boolean	If true and HJ occurs during this turn, ignore it (rather than interrupt the turn). (default: false)
ignoreGlobalRules	boolean	If true, global rules are not active during the turn, if false, the global rules (other than the launch rule) are active in addition to the provided nluRules. (default: false)
rejectIfBusy	boolean	If true and there is already a turn in progress (global, local, enrollment or pronunciation learning) then reject this turn, otherwise interrupt the turn in progress and run this one instead. (default:false)
language	string	ISO language label, e.g. "en-US" (default: "")
meta	string	Arbitrary JSON data to pass along to the hub. (default: "")
earlyEOS	string[]	A list of words and phrases that will be used to detect a valid response from incremental ASR results (i.e. prior to EOS and grammar parsing).
hintPhrases	string[]	A list of hint phrases to add to the ASR vocabulary
clientNLU	NLUResult	The semantic result desired from this turn. If present, no audio will be sent to the hub. See this entry in update_local_turn for further details.
clientASR	string	The ASR text result desired from this turn. If present, no audio will be sent to the hub. See this entry in update_local_turn for further details.
sosTimeout	float	The period of time to wait (in seconds) for the Start Of Speech after hearing HJ. If HJ is not heard within this period, the turn completes with a status of TIMEDOUT and message of "sos". If this property is set to -1 (or not provided) the value used is the one found in the jibo-jetstream-service.json file with the property name "local_sosTimeout_sec", which is currently 10 seconds. (default: -1)
maxSpeechTimeout	float	The maximum number of seconds that a user may speak after saying HJ. If the user speaks for more than this number of seconds, the turn completes with a status of TIMEDOUT and message of "maxSpeech". As with sosTimeout, a value of -1 causes the config file property "local_maxSpeechTimeout_sec" to be used, which is currently 60 seconds. (default: -1)
Response
name	type	Description
requestID	string	A unique ID for this turn, which will be included with any status information about it posted to `/events` and which can later be used to cancel it.

Update Local Turn

Update (that is, force the completion of) an in-progress local turn with the given NLU or ASR text information. This function is provided to allow a menu button push to simulate NLU input to the skill. When invoked, the audio being streamed to the Hub is stopped and the Hub accepts the provided ASR or NLU data as input.

There is a potential race condition where the given ASR/NLU data is sent to the Hub but the Hub (which is also waiting for results from the Cloud ASR and Cloud Parser) receives the Cloud ASR or NLU result first, which causes the update command to be ignored.

URL

/listen/update_local_turn

Parameters

name

type

Description

requestID

string

The requestID of the in-progress turn to be updated. If the turn for the given requestID is no longer active, this call is ignored.

Mimic Global Turn

Similar to Update Local Turn, the skill provides ASR text or NLU values with which to simulate a global turn (i.e. as if the user had said "HJ" followed by the given ASR text or an utterance that resulted in the given NLU values). This function is provided to allow cloud skills to be launched from the robot.

URL

/listen/mimic_global_turn

Parameters

name

type

Description

language string ISO language label, e.g. "en-US"

suppressedEvents string[] A list of events to suppress during this turn, e.g. ["HJ_HEARD", "TURN_STARTED"]. All events that can be emitted during a global turn can be suppressed. And note that only those events associated with this global turn can be suppressed. If, for example, someone says HJ during a mimic_global_turn, that HJ_HEARD event will not be suppressed, even if HJ_HEARD is among the events to be suppressed during this turn.

clientNLU

NLUResult

The semantic result desired from this turn. This will be the same JSON schema that arrives as the data.result.nlu field in a successful TURN_RESULT message, e.g.

"nlu" : {
 "entities" : {},
 "intent" : "generalWhatQuestions",
 "rules" : [
   "launch"
 ]
}

NLUResult
name	type	description
rules	string[]	listen rules to activate
intent	string	intent
entities	object	a map of EntityNames (strings) to EntityValues (which are either string or string[])

clientASR string The ASR text to be used to produce the NLU result from this turn. This is a simple string of words, equivalent to what the ASR returns. It should not include the "Hey Jibo" text. Only one of the clientASR and clientNLU properties should be provided, if both are provided only the clientNLU property will be used.

Response

name

type

Description

requestID

string

A unique ID for this turn, which will be included with any status information about it posted to /events and which can later be used to cancel it.

Cancelling Turns

Cancel Local Turn and Cancel Any Turn allow turns currently in progress to be cancelled. When cancelled, a status message will appear on /events indicating the turn was cancelled. Enrollment and Pronunciation Learning turns are also cancelled this way. If there was no turn in progress (or if it completed before the cancel was executed) then the cancel request is ignored and no status is given.

Cancel Local Turn

Cancel the turn matching the given requestID.

URL	/listen/cancel_local_turn
Parameters	name	type	Description
	requestID	string	The requestID of the turn to cancel
Response	NONE (empty JSON)

Cancel Any Turn

Cancel any turn that is in progress, including a global turn (one started by the user uttering "HJ").

URL	/listen/cancel_any_turn
Parameters	NONE (empty JSON)
Response	NONE (empty JSON)

Global Subscriptions: Handling "HJ + phrase"

Grammars for handling user speech of the form "HJ+phrase" are installed and removed through Subscribe and Unsubscribe requests. Normally the set of grammars used for any particular "HJ+phrase" is the union of the grammars of all the active subscriptions. However, the "exclusive" parameter, when set, will cause only those subscriptions so marked to be active (i.e. the subscriptions not marked exclusive will be hidden by the ones marked exclusive).

Subscribe Global

Create a subscription that contributes the given grammars to the set of all active global grammars.

URL	/listen/subscribe_global
Parameters	name	type	Description
	exclusive	boolean	When true, only the subscriptions marked exclusive will be active.
	nluRules	string[]	A list of grammars to be added to the set of grammars with which to recognize HJ+phrase requests
	language	string	an ISO language tag, e.g. "en-US"
Response	name	type	Description
	requestID	string	A unique ID for this subscription, which can be used to later unsubscribe it.

Unsubscribe Global

Remove a particular subscription, given by its ID, from the set of active subscriptions.

URL	/listen/unsubscribe_global
Parameters	name	type	Description
	requestID	string	A ID of the subscription to be removed.
Response	NONE (empty JSON)

Unsubscribe All Globals

Removes all subscriptions.

URL	/listen/unsubscribe_all_globals
Parameters	NONE (empty JSON)
Response	NONE (empty JSON)

Speaker Voice Enrollment

Speaker voice enrollment consists of creating models of each speaker's voice by asking them to say HJ some number of times. In order for the skill to control this process there are commands for removing existing models, collecting HJ samples (aka "utterances"), and creating models from the previously collected samples.

The usual workflow for a skill using the enrollment API is to start an enrollment turn for a particular speaker that collects their utterances. The skill can request any number of utterances, but it would typically be six (or just a single one if recreating the Jibo V1 UX).

Once the turn begins (indicated by a TURN_STARTED event) the speaker says each HJ separated by at least a second. After each HJ an event is emitted reporting the utterance quality and the utterance number (out of the total requested) that this represents.

When the requested number of acceptable HJs have been collected, a TURN_RESULT event is emitted (status=SUCCEEDED) and the enrollment turn is finished.

Even if an enrollment turn is cancelled or interrupted, the acceptable utterances collected so far for each speaker remain as "pending" utterances in the system. The /enroll/get_utterance_count method may be used to find out how many utterances a speaker has, both those for use in creating a new model (pending utterances) and of those that were used to create the existing model (model utterances).

Finally, the skill requests that a new model be created for a given speaker from all the utterances that have been collected for them and (optionally) the utterances used to create the existing model. This returns a status of either SUCCEEDED or FAILED.

Start Enrollment Turn

This is the speech sample collection step for voice enrollment. It is designed so that the user can say several HJs in succession. Every time an HJ is said, a message is posted to /events giving the status of the utterance (i.e. whether it was acceptable or not). After the requested number of (acceptable) HJs is given, the turn will complete with a result of SUCCEEDED. Acceptable utterances collected during an enrollment turn are collected into a set of pending utterances that exist until a model is created from them.

In the case of a catastrophic failure of the enrollment recognizer the TURN_RESULT message will have a status of FAILED and the data.message field will contain the reason for the failure.

URL	/listen/start_enrollment_turn
Parameters	name	type	Description
	numberOfUtterances	number	The number of (good) HJs the user must say before the turn reports SUCCEEDED
	speakerID	string	An arbitrary string that the skill provides to denote a particular speaker. This string will be used to identify the utterances collected here when the speaker model is created.
	rejectIfBusy	boolean	If true and there is already a turn in progress (global, local, enrollment or pronunciation learning) then reject this turn, otherwise interrupt the turn in progress and run this one instead.
Response	name	type	Description
	requestID	string	A unique ID for this turn, which will be included with any status information about it posted to `/events` and which can later be used to cancel it (via cancel_local_turn)

Create Speaker Model

This creates a speaker mode from a) the pending samples collected during the enrollment turn(s) for a particular speaker and –if append is true– b) the model samples used in the creation of the previous model.

The SpeakerID provided can refer to any speaker from whom a sufficient number of samples is available (i.e. the sum of those pending and –if append is true– those already in the model). Currently at least 4 samples a required from any particular speaker.

After the model is created, the pending samples used to create the model are moved into the so-called model category to join (append=true) or replace (append=false) the other model utterances there, if any. In this way the model utterances always represent the set of utterances used to build the most recently created model. By setting append to true in subsequent model creation operations, the samples collected then will be used together with (i.e. appended to) all the ones used to make the current model.

URL	/enroll/create_speaker_model
Parameters	name	type	Description
	speakerID	string	The ID of the speaker whose model is being created (the same value used for the speaker's enrollment turns).
	append	bool	When true, append the pending utterances to the model utterances and use both sets to create the model. When false, only the pending utterances are used to create the model. Default: false
Response/Code	name	type	Description
HTTP 200	Empty JSON
HTTP 400	message	string	Error message (e.g. "Insufficient Utterances")

Remove Speaker Model

This removes the speaker model and all the model utterances for a given speaker. The speaker is effectively removed from the Speaker ID system. The pending utterances for a given speaker are not affected (they can be removed with /enroll/remove_pending_samples)

URL	/enroll/remove_speaker_model
Parameters	name	type	Description
	speakerID	string	The speaker ID for the model to be removed.
Response/Code	name	type	Description
HTTP 200	Empty JSON
HTTP 400	message	string	Error message (e.g. "No Such Speaker")

Remove Pending Samples

This removes the pending utterances collected for a given speaker (i.e those not yet made into a model). The speaker model and the model utterances are not affected (they can be removed with /enroll/remove_speaker_model)

URL	/enroll/remove_pending_samples
Parameters	name	type	Description
	speakerID	string	The speaker ID for the pending samples to be removed.
Response/Code	name	type	Description
HTTP 200	Empty JSON
HTTP 400	message	string	Error message (e.g. "No Such Speaker")

Get Utterance Count

This returns the number of good utterances collected so far for a given speaker (but not yet built into a model, the so-called pending_utterances) and the number of utterances used in the existing model for the speaker (the so-called model_utterances).

URL	/enroll/get_utterance_count
Parameters	name	type	Description
	speakerID	string	The ID of the speaker for whom the count of utterances is requested.
Response	name	type	Description
	pending_utterances	number	The number of utterances that have been collected so far for the given speaker but not yet built into a model.
	model_utterances	number	The number of utterances that were used to build the current model.

Get Enrolled Speakers

This returns the speaker IDs for all the speakers for which models have been created.

URL	/enroll/get_enrolled_speakers
Parameters	NONE (empty JSON)
Response	name	type	Description
	speakers	string[]	A list of the speaker IDs of the enrolled speakers.

Pronunciation Learning

Also called "name learning", Jetstream provides methods for finding the user's pronunciation of a name. The workflow for skills consists of just two steps: initializing the system with the name to be learned and then asking the system to listen to the user say the name and return an estimate of the way it was pronounced.

Initialize Pronunciation Learning

This prepares the system for listening to the way a user pronounces a particular word. It builds data structures that will be used during the listening step. For normal length names this completes in under 50ms, but longer names can take several hundred ms.

URL	/pronunciation/init_pronunciation_learning
Parameters	name	type	Description
	word_to_learn	string	The word that the system will try to learn the pronunciation of.
Response	name	type	Description
	requestID	string	A unique ID that is created every time `init_pronunciation_learning` is called that is included in messages and audio files logged during subsequent pronunciation learning turns. Its sole purpose is to enable post-hoc speech science experiments by allowing the initialization of the pronunciation learning system and the audio data logged in subsequent turns to be tied together.

Start Pronunciation Learning Turn

This is the pronunciation estimation step where the user speaks the name to be learned. It first processes the incoming audio with a speech endpointer to determine when the user is speaking. Then, constrained by a phonetic graph built from the spelling of the word, it asks a recognizer to return the best path through that graph given what was said. The result is a (probabilistic) phonetic spelling of what it heard the user say.

URL	/listen/start_pronunciation_learning_turn
Parameters	name	type	Description
	word_to_learn	string	This must be identical to the word provided in the init_pronunciation_learning step. If not, an error is returned.
	ignoreHJ	boolean	If true, any utterances of HJ will be ignored during this turn. If false, HJ will have its normal effect (usually interrupting the turn in progress and starting a global turn). (default: false)
	rejectIfBusy	boolean	If true and there is already a turn in progress (global, local, enrollment or pronunciation learning) then reject this turn, otherwise interrupt the turn in progress and run this one instead. (default: false)
Response	name	type	Description
	requestID	string	A unique ID for this turn, which will be included with any status information about it posted to `/events` and which can later be used to cancel it (via cancel_local_turn)

Altering how HJ is handled

Normally when a user says HJ, Jetstream will interrupt whatever is in progress and begin a "Global Turn", sending the audio following the HJ to the Hub for speech-to-text and parsing, and then returning a possible skill switch directive from the Hub if what has been said matches one of the active global grammars.

However, Jetstream can be set to handle HJ in other ways by using Set HJ Mode. When this is done the new setting stays in effect until the robot is rebooted or the setting is changed again.

Set HJ Mode

This sets the HJ handling mode. There are 3 ways of handling HJ in Jetstream:

"NORMAL_HJ", as described above, handles HJ by starting a global turn.
"IGNORE_HJ" causes Jetstream to ignore any utterances of HJ; no events are emitted and there are no changes to Jetstream's state, it is as if the user has not said HJ at all.
"ONLY_HJ" has the effect of letting Jetstream recognize HJ and emit an event indicating such, but it does not start a global turn and instead simply returns to the base state of waiting for the next HJ. Note that if a turn (global, local, etc) is in progress, an HJ in this mode will still have the effect of interrupting that turn just as it would in NORMAL_HJ mode.

URL	/listen/set_hj_mode
Parameters	name	type	Description
	mode	string	One of three possible values: NORMAL_HJ ONLY_HJ IGNORE_HJ
Response	None (empty json)

Get HJ Mode

This is used to discover Jetstream's current HJ handling mode. Note that because of the way Jetstream works it is possible to set the mode and then get the mode within a short enough interval (<10 milliseconds) that the change has not had time to take effect and in this case the old mode will be reported.

URL	/listen/get_hj_mode
Parameters	None (empty json)
Response	name	type	Description
	mode	string	The current setting of Jetstream's HJ handling mode.

Proactive Triggering

Proactive Triggering is the way that Jibo "wakes up" the Hub to find out whether there is some interaction it can offer the user. Since the Hub cannot contact Jibo, it is up to Jibo to contact the Hub.

Trigger

Trigger does not take any JSON parameters itself, rather the entire body of the HTTP request is sent, verbatim, to the Hub. Internally Trigger does request the Jibo Context from Be and sends that to the Hub right after the information provided in the Trigger request.

URL	/proactive/trigger
Parameters	None – the contents of the HTTP request's body is passed directly to the Hub
Response	name	type	Description
	requestID	string	A unique ID for this trigger, which will be included with Trigger response messages posted to `/events.`

Test and Debug Commands

To enable test and debug efforts, Jetstream offers a few commands for recording audio from Jibo's "ASRin" audio source or inserting audio from an external file so that Jetstream uses it in lieu of microphone input.

Start Recording

This makes a recording from Jibo's "ASRin" audio device, which is actually the output of the Jibo AudioService (the Audio Service provides a single, noise-cancelled, virtual audio source derived from Jibo’s six individual microphones). The audio's sampling rate is 16kHz with 16 bit, little-endian samples. While recording, the audio is stored in a file. When the Retrieve Recording command is issued, the recording process is stopped and the recorded audio is returned.

URL	/debug/start_recording
Parameters	name	type	Description
	start_immediately	boolean	When true, recording will begin as soon as the command is issued, when false, recording will begin when HJ is recognized.
	max_recording_duration_ms	number	Sets the maximum length of time (in ms) that the recording will proceed. Default is 2000 ms.

Retrieve Recording

This stops the recording (if in progress) and returns the audio and cue markers from the recording.

URL

 /debug/retrieve_recording

Parameters

None (empty json)

Response

name

type

description

recording

RetrievedRecording

A JSON object containing the base64 encoded audio of the recording and cue markers, as follows:

Retrieved Recording
name	type	description

base64_audio	string	binary audio data in base64 format.
start_time	number	always 0
end_time	number	duration of the recording in ms
cue_points	CuePoint[]	an array of CuePoints, described below.

CuePoint
name	type	description

name	string	the name of the point, one of HJ_BEG, HJ_END, EP_BEG, or EP_END
time	number	the time point (number of milliseconds into the recording) where the cue point occurs.

Note that the Jetstream repo includes a "record-from-robot.py" utilitiy for making robot recordings from the Mac that places the recording's cue points into the wave file it creates. These cue points will be displayed as labeled markers in Mac audio editors like "TwistedWave":

Mimic Mic Input

For testing purposes this command allows you to temporarily replace Jibo's normal microphone audio input with audio data you supply. The supplied data is fed to the robot at the same rate as normal audio input, so (in theory) this should be indistinguishable from normal audio input. Problems arise however when using TTS synthesized audio since the background noise from the microphone is quite different from the background noise of TTS (which is essentially zero noise) and the switching between the two noise levels when the external audio is introduced and removed causes recognition problems. In practice we have found that noise must be mixed with TTS audio for the best recognition results. Our noise source was simply a recording made from Jibo in a quiet room.

Use of this command while audio from a previous command is still being used simply results in the new audio being queued for use after the existing audio finishes. In addition to the audio, it is possible to specify pre- and post- insertion delay times that will cause a delay before the next audio in the queue is used (microphone input is used during the delay period). This can be useful for experimenting with the size of the gap in e.g. "HJ <gap> <phrase>" or other speech strings, without having to make new recordings.

URL	/debug/mimic_mic_input
Parameters	name	type	Description
	audio	string	the base64 encoded audio data at 16kHz and 16 bits per sample.
	start_delay_ms	number	the number of ms to wait before using the supplied audio
	finish_delay_ms	number	the number of ms to wait after using the supplied audio before using the next audio in the queue.

Stop Mimicking Input

This command will stop any currently active mimicked input. It returns the sample number of the last sample given to the robot's audio system. This allows you to find the spot where playing left off in the file that was presented Note that any queued "mimicked input" is not affected and after this command runs, the next item in the queue (if any) will be feed to the audio system.

URL	/debug/stop_mimicked_input
Parameters	None (empty json)
Response	name	type	Description
	stopping_point	number	the sample number at which audio input to the robot stopped (the first sample in the file that was not sent to the robot – accurate to ~50ms)

The Jetstream Websocket API

Jetstream reports the status and conclusion of most of its commands via Websocket messages through the URL /events. The Jetstream Service Debug page (http://172.24.84.101:8090/index.html) displays the messages being sent to /events as well as messages from several other debug-related URLs.

Jetstream does not accept any input on its websocket ports and will ignore what is sent to them.

WSmsg: the base class

All Jetstream WS messages have these fields in common.

Name	(base class)
Parameters	name	type	Description
	type	string	the name of the message e.g. "TURN_RESULT"
	requestID	string	if this is in response to a command, this will be the requestID Jetstream returned when the command was issued.
	transID	string	if this is related to a Hub transaction, this is the transaction ID Jetstream used with the Hub during that transaction.
	ts	number	the time in milliseconds since the Unix epoch
	data	object	Jetstream messages that need to provide information beyond the base class properties populate a "data" object with the information. The data object schema is message dependent.

Turn-related Messages

During the processing of a turn, these messages can appear.

Turn Started

This message is sent as a status update for all turns (local, global, enrollment, and pronunciation learning). For local and global turns this indicates that a connection to the Hub has been successfully established and that audio is being sent. For enrollment and pronunciation learning turns, this indicates that the underlying recognizer has been engaged and is processing user audio.

Name	TURN_STARTED
Parameters	NONE

Start of Speech

Upon receiving a Hub notification that its speech recognizer has reported Start of Speech, this message is sent.

Name	SOS
Parameters	NONE

End of Speech

Upon receiving a Hub notification that its speech recognizer has reported End of Speech, this message is sent. In response to that Hub notification, Jetstream stops sending audio to it.

Name	EOS
Parameters	NONE

Turn Result

This message is sent at the conclusion of a turn and the status field contains the reason it concluded. In the case where the status is SUCCEEDED, the recognition result from the turn is also provided, in the case where the status is FAILED, the error message giving the failure reason is provided. In the case of TIMEDOUT the message field will be either "sos" or "maxSpeech". In all other status cases no further information is provided. All turns are guaranteed to conclude with a Turn Result message of some sort.

Name	TURN_RESULT
Parameters	name	type	Description
	data.status	string	The reason the turn concluded, one of: SUCCEEDED FAILED INTERRUPTED (by another turn request or HJ+phrase) CANCELED (by cancel_local_turn) REJECTED (when a turn is started with `rejectIfBusy` set to true and another turn is already in progress) TIMEDOUT (when user speech does not occur within the period specified by sosTimeout or when the duration of user speech exceeds the maxSpeechTimeout setting)
	data.message	string	(If present) In the case of status==FAILED or TIMEDOUT this contains the reason for the failure or timeout.
	data.message	string	(If present) In the case of status==SUCCEEDED and a Pronunciation Learning turn, this contains the estimated pronunciation of what the user said.
	data.result	object	(If present) In the case of status==SUCCEEDED (and not a Pronunciation Learning turn) this contains the "result" object from the Hub.

Hub-related Messages

Hub Error

Hub Error messages cover a lot of ground: Jetstream itself can produce them in response to Proactivity Trigger requests if it fails to open a Hub connection or if an already open Hub connection goes down. Or the Hub can ask them to be produced as a response to a previous request (e.g. LISTEN or TRIGGER).

In the case of Jetstream-sourced messages only the "data.message" parameter is present, which gives the reason for the failure. In the case of a Hub-promulgated message, the data object is a verbatim copy of the JSON field the Hub supplied.

Name	HUB_ERROR
Parameters	name	type	Description
	data.message	string	(If present) The problem Jetstream encountered trying to communicate with the Hub.
	data	object	(If present) The Hub's error information it supplied in response to a LISTEN or TRIGGER request.

SKILL_*

There are a class of messages the Hub produces that start with the string "SKILL_". These are directly transcribed into Jetstream messages containing a verbatim copy of the JSON from the original Hub message.

Name	SKILL_*
Parameters	name	type	Description
	data	object	The JSON the Hub supplied.

Onboard Recognizer-related Messages

These messages correspond to events that occur in the onboard recognizers.

HJ Heard

HJ Heard is emitted as soon as the onboard recognizer is confident it has heard someone say HJ, this may actually happen before the "b" in Jibo.

Name	HJ_HEARD
Parameters	NONE

HJ Only

HJ Only is emitted when the Hub reports that no speech after the HJ was detected or recognized. The delay between the time the HJ was said and this event is emitted is determined by the HubAsr.global_sosTimeout_sec setting in the config file.

Name	HJ_ONLY
Parameters	NONE

Speaker ID

Speaker ID is emitted only if there are enrolled speakers.

Name

SPEAKER_ID

Parameters

name

type

Description

data.speakers

SpeakerRecogResult[]

An array of SpeakerRecogResult objects (sorted in descending score order) with one entry for each enrolled speaker that states how well that speaker was recognized.

SpeakerRecogResult
name	type	description
speaker	string	The ID string used to enroll this speaker.
score	number	authentication score (not useful without knowing the threshold)
accepted	boolean	If true, this speaker's score was high enough to be considered "recognized"
high_confidence	boolean	If true, this speaker's score was either high enough that the system is confident in accepting it, or low enough that it is confident in rejecting it.

data.snr

number

The estimated signal-to-noise ratio of the HJ input used in the speaker recognition operation.

Speaker Enrollment

A Speaker Enrollment message is emitted after every HJ recognition during an enrollment turn.

Name	SPEAKER_ENROLLMENT
Parameters	name	type	Description
	data.speaker	string	The ID of the speaker being enrolled.
	data.snr	number	The estimated SNR of the HJ utterance.
	data.accepted	boolean	True if the utterance quality was acceptable. If this is false, the utterance being reported on was rejected.
	data.problems	string[]	If data.accepted is false, this is a string vector containing the problem(s) with the utterance. The possible values are: CLIPPING, POOR_SNR, NOT_SPEECH_LIKE, NOT_CONSISTENT, BAD_DURATION, BAND_LIMITED, POOR_QUALITY.
	data.good_utterance_count	number	The number of utterances accepted so far during this turn.
	data.total_utterance_count	number	The total number of utterances seen so far during this turn.
	data.final	boolean	True if the number of acceptable utterances matches the numberOfUtterances parameter specified when the enrollment turn was started. After this the turn will complete and no further Speaker Enrollment messages will appear.

Voice Activity Detector events

Jetstream has a built-in voice activity detector that emits websocket messages at 20hz to the endpoint /vad. These messages indicate the degree to which the sound that Jibo's microphones are picking up resembles human speech.

Name VAD_EVENT_DATA

Parameter Type Description

audio_time

uint64_t

The system time corresponding to the beginning of the results

avg_frame_db float The average power of the 50ms frame of audio that was used to produce the VAD results.

VADevents

VADevent[]

An array of VADevent elements. Each corresponds to a 20ms span of audio. Since the VAD uses a 20ms frame internally and Jetstream uses a 50ms frame (that is, always giving the VAD 50ms of audio at a time), the length of the VADevents array will alternate between being two and three elements long.

VADevent
name	type	description
time	uint64_t	The system time corresponding to the audio that produced this result.
vad	float	This is the probability that there is voice in each 20ms of audio. These probabilities are float values from approx. -1.0 to approx. +1.0, where 0.0 can be seen as the recommended threshold for the no-voice (<0) or voice (>0) decision. To lower sensitivity, increase the threshold (and vice versa). Sensible ranges for a threshold are between -0.3 and + 0.4. The closer the values are to -1.0 and +1.0, the more certain the decision as to whether the audio is voice or not-voice.

Jetstream's Debugging and Testing Facilities

The `/debug/*` Websocket ports

As described in the prior section, Jetstream provides information to skills via the /events websocket. To assist debugging and testing Jetstream also produces WS events on 3 other ports, The output from these ports is visible on the Jetstream debug page. We describe them below.

The Listener Loop monitor: `/debug/listenloop`

This port shows all the events being sent to the Listener Loop (Jetstream's central process).

Looking on the Jetstream debug page while it is running, we see output like this in the listen_loop window:

{"state":"hjw","ts":1519834943931,"type":"JM_RECOG_HJ_EVENT","data":{"name":"RecogHJEventData","timestamp":1519834943778,"begin_time":945765,"end_time":946435,"max_frame_power":2.431922674179077,"utterances":["hey jibo"],"scores":[110.0],"message":" utterance: hey jibo score: 110 walign: ADE00D70 palign: ADE00C60 {\"word_alignment\":\"945765 945960 hey 0.00 945960 946185 jibo 0.00 \"}"}}
{"state":"phw","ts":1519834943943,"type":"JM_HUB_CLIENT_CONNECT_RESULT","transaction_id":"tid:8f021c76-1ca3-11e8-9f1f-8cae4cf576b1","success":true,"message":""}
{"state":"phw","ts":1519834944148,"type":"JM_RECOG_SPEAKER_EVENT","data":{"speakers":[{"speaker":"5a285d26316a8a00193a0e7c","score":5.6092047691345219,"accepted":true,"high_confidence":true}],"snr":18.92803955078125}}
{"state":"phw","ts":1519834945141,"type":"JM_HUB_SOS","transaction_id":"tid:8f021c76-1ca3-11e8-9f1f-8cae4cf576b1"}
{"state":"phw","ts":1519834946618,"type":"JM_HUB_EOS","transaction_id":"tid:8f021c76-1ca3-11e8-9f1f-8cae4cf576b1"}
{"state":"phw","ts":1519834947001,"type":"JM_HUB_LISTEN","transaction_id":"tid:8f021c76-1ca3-11e8-9f1f-8cae4cf576b1","final":false,"data":"{ \"asr\" : { \"confidence\" : 0.9876290559768677, \"text\" : \"do you like penguins\" }, \"match\" : { \"onRobot\" : false, \"skillID\" : \"chitchat-skill\" }, \"nlu\" : { \"entities\" : { \"Activity\" : \"\", \"GeneralLikes\" : \"Penguin\", \"Occupation\" : \"\", \"Other\" : \"\", \"OtherPerson\" : \"\", \"given-name\" : \"\", \"last-name\" : \"\", \"number\" : \"\" }, \"intent\" : \"doesJiboLikeThing\", \"rules\" : [ \"launch\", \"globals/global_commands_launch\" ] } }"}
{"state":"phw","ts":1519834947112,"type":"SKILL_ACTION","transaction_id":"tid:8f021c76-1ca3-11e8-9f1f-8cae4cf576b1","type":"SKILL_ACTION","final":true,"data":"{ \"action\" : { \"config\" : { \"es_auto_tagging\" : true, \"mim_type\" : \"announcement\", \"prompts\" : [ { \"prompt\" : \"Yes I love penguins, because we're so alike. We have the same coloring, and neither of us can fly.\", \"prompt_id\" : \"RI_JBO_LikesPenguins_AN_04\" } ] }, \"type\" : \"SLIM\", \"version\" : \"1.0.0\" }, \"final\" : false, \"fireAndForget\" : false, \"skill\" : { \"id\" : \"chitchat-skill\", \"session\" : { \"data\" : { \"_mim\" : { \"noInput\" : 0, \"noInputMax\" : false, \"noMatch\" : 0, \"noMatchMax\" : false } }, \"id\" : \"be2ab9c3-1c94-4553-9364-59961c0b70df\", \"nodeID\" : 0, \"trace\" : [ { \"nodeID\" : 1, \"transition\" : \"Reactive\" }, { \"nodeID\" : 2, \"transition\" : \"ScriptedResponse\" }, { \"nodeID\" : 0, \"transition\" : null } ] } } }"}

Each line is the JSON representation of a message that the Listener Loop has received. The state the Listener Loop was in when the message was received is placed in the "state" property (hjw: HJ Wait, phw: Phrase Wait, plw: Pronunciation Learning Wait, enw: Enrollment Wait).

In the list above we can see how an utterance of "Hey Jibo, do you like penguins?" has become 7 events submitted to the Listener Loop:

The HJ recognizer seeing "HJ"
The Hub reporting that it is connected
The Speaker ID recognizer reporting a recognition event
The hub reporting SOS
The hub reporting EOS
The hub reporting LISTEN results from the cloud ASR, and finally
A hub skill switch event based on a parse of what was said.

Monitoring the timestamps here can show how long these events are taking to arrive.

The Hub monitors: `/debug/hub_outgoing` and `/debug/hub_incoming`

These two ports log all the messages Jetstream is sending to the Hub and (respectively) that the Hub is sending to Jetstream. This is what they produced during the same interaction above:

hub_outgoing (from Jetstream to the Hub)

{"type":"LISTEN","ts":1514742224646,"msgID":"mid:25c4afe6-ee52-11e7-aad4-8ace4cffad67","transID":"tid:25bf5e7e-ee52-11e7-be44-8ace4cffad67","data":{"lang":"en-US","hotphrase":true,"rules":["launch"]}}
{"type":"CONTEXT","ts":1514742224647,"msgID":"mid:25c4db56-ee52-11e7-8c29-8ace4cffad67","transID":"tid:25bf5e7e-ee52-11e7-be44-8ace4cffad67","data":{"general":null,"runtime":null}}

The LISTEN message indicates to the Hub that recognition is requested. It is followed by a CONTEXT message holding information about the state of the robot obtained from the Be "Context" service. In this case the service was not running, so the runtime field is "null".

After the LISTEN and CONTEXT messages, the audio will begin being sent. The audio messages are of type binary (which is how the hub distinguishes them from non-audio messages) and are not sent to the hub_outgoing port.

hub_incoming (from the Hub to Jetstream)

{"type":"SOS","data":null,"msgID":"c78a434e-fd2c-4c9b-bb1c-8b47d3fc3110","transID":"tid:25bf5e7e-ee52-11e7-be44-8ace4cffad67","ts":1514742225689,"timings":{"total":1061}}
{"type":"EOS","data":null,"msgID":"50454cc9-9285-4530-b7df-f2ec1bf141d8","transID":"tid:25bf5e7e-ee52-11e7-be44-8ace4cffad67","ts":1514742226861,"timings":{"total":2233}}
{"type":"LISTEN","msgID":"296a7c5a-5e97-4f31-8e4d-d3faec360410","transID":"tid:25bf5e7e-ee52-11e7-be44-8ace4cffad67","ts":1514742227286,"data":{"asr":{"text":"do you like penguins","confidence":0.9797691106796265},"nlu":{"rules":["launch"],"intent":"doesJiboLikeThing","entities":{"FoodGeneral":"","Mammal":"","Amphibian":"","Bird":"Penguin","Breakfast":"","Bug":"","Cheese":"","Color":"","Dinosaur":"","FantasyCreature":"","Fish":"","Flower":"","Fruit":"","MusicType":"","MusicalInstrument":"","Pet":"","SchoolSubject":"","Season":"","Shape":"","Sport":""}},"match":{"skillID":"chitchat-skill"}},"final":false,"timings":{"total":2658,"asr":2364,"nlu":291}}
{"type":"SKILL_ACTION","data":{"skill":{"id":"Chitchat","session":{"id":"110bd7fd-4419-49a5-b00c-3aeab05325aa","nodeID":0,"data":{"_mim":{"noMatch":0,"noInput":0,"noMatchMax":false,"noInputMax":false}},"trace":[{"nodeID":1,"transition":"Reactive"},{"nodeID":2,"transition":"ScriptedResponse"},{"nodeID":0,"transition":null}]}},"action":{"version":"1.0.0","type":"SLIM","config":{"prompts":[{"prompt_id":"RI_JBO_LikesPenguins_AN_03","prompt":"I like lots of animals, but the penguin is the best of the best. <break size='.3'/> Great color scheme."}],"mim_type":"announcement"}},"final":false,"fireAndForget":false},"ts":1514742227296,"transID":"tid:25bf5e7e-ee52-11e7-be44-8ace4cffad67","msgID":"a2abdd8e-0b6f-4d06-bff7-22b4b7b5d42c","timings":{"total":2670,"skill":10},"final":true}

The timestamps in the messages appearing in hub_incoming are generated on the Hub, so a way to get a sense of the time skew between the two systems is to look at e.g. the difference between the timestamp on the SOS message from he Hub and the one on the (related) SOS message being posted to the Listener Loop. In this case they report being 20ms apart (1514742225709 (Jetstream message) - 1514742225689 (Hub msg) = 20).

Jetstream Flow Diagram

Jetstream was originally developed from the following diagram showing how Local and Global turns are handled. Further functions and turn types have since been added to Jetstream (i.e. Enrollment and Pronunciation Learning), but the basic structure still holds and serves as a useful sketch of its operation.

Source: link

Summary

And this concludes our tour of the Jetstream system.

Comments and corrections are welcome.

Page tree

The Jetstream Service API

Service Description

The Jetstream HTTP API

Local Turns: Skill-driven ASR

Start Local Turn

Update Local Turn

Mimic Global Turn

Cancelling Turns

Cancel Local Turn

Cancel Any Turn

Global Subscriptions: Handling "HJ + phrase"

Subscribe Global

Unsubscribe Global

Unsubscribe All Globals

Speaker Voice Enrollment

Start Enrollment Turn

Create Speaker Model

Remove Speaker Model

Remove Pending Samples

Get Utterance Count

Get Enrolled Speakers

Pronunciation Learning

Initialize Pronunciation Learning

Start Pronunciation Learning Turn

Altering how HJ is handled

Set HJ Mode

Get HJ Mode

Proactive Triggering

Trigger

Test and Debug Commands

Start Recording

Retrieve Recording

Mimic Mic Input

Stop Mimicking Input

The Jetstream Websocket API

WSmsg: the base class

Turn-related Messages

Turn Started

Start of Speech

End of Speech

Turn Result

Hub-related Messages

Hub Error

SKILL_*

Onboard Recognizer-related Messages

HJ Heard

HJ Only

Speaker ID

Speaker Enrollment

Voice Activity Detector events

Jetstream's Debugging and Testing Facilities

The /debug/* Websocket ports

The Listener Loop monitor: /debug/listenloop

The Hub monitors: /debug/hub_outgoing and /debug/hub_incoming

hub_outgoing (from Jetstream to the Hub)

hub_incoming (from the Hub to Jetstream)

Jetstream Flow Diagram

Summary

The `/debug/*` Websocket ports

The Listener Loop monitor: `/debug/listenloop`

The Hub monitors: `/debug/hub_outgoing` and `/debug/hub_incoming`